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ABSTRACI}^ . - 

The pamphlet describes an evaluation ©odel^ for. crime . 
reduction projects gleveloped* by National Institute of law Enforcement 
and Criminal Justice sponsored res*earch. It is meant to assist 
justice^ agency and project managers in determining completeness of 
evaluation planning by providing a framework against which to measure 
their evaluation componentsWwhile the model and review criteria 
presented were developed within the context of the Legal Enforcement 
Assistance Administration^ s (LEAA's) High Impact Anti^-Crime Programr 
'they we^e based upon an awareness af the difficulties involved in. 
evaluating social programs generally and maj be applicable to - 
evaluation efforts outside of^'-the criminal justice sector. The paper 
is divided into IToHr s§ctians. The first (introductory) sectioja 
describes current preoccupations with ev^'luation, Th% second Section 
provides the reader with an understanding of the special context 
within . which the model and 'criteria were developed via a brief 
discussion of the Impact Program's evaluation effort. The third 
section " presents the evaluation plannl/ng model along with a*' . 
discussion of key steps in the evaluation-' planning process , The 
fourth. section elaborates general guidelines regarding the use and 
applicability . of the model and review criteria, and develop^ a ^set of 
questions whiph need -to be- addressed during the teview of a 
pro ject--level evaluation pl^n or componB^t, (Aiithor/BJG) 
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FOREWORD 

Evaluating social programs is a difFtcult task. Given the unknowns in the 
social science field and the modesty of the evaluation tools currently at our 
•disposal, answers often are beyond readi. This is especially true in criminal 
justice, a field in which the kind of planning and data gathering necessary 
for evaluation has only recently begun. 

Despite these obstacles, evaluation of the ijiipact of LEAA-funded 
programs is essential. The National Institute is giving high priority to this 
task and to building state and local evaluation capabilities. 

The pamphlet describes an' evaluation model for crime reduction projects 
developed by National Institute sponsored research. It is being distributed 
to assist criminal justice^ agency and project managers in determining tiie 
completeness of evaluation planning by providing a framework against 
which to measure their evaluation components. 

Gerald M. Caplan, r 
Director 

: National Institute of Law ^ 

Hnforcement and Criminal Justice 
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PREFACE 

As pact ofUie national leve^eyall^atiqn of the LEAA's High Impact Anti- 
Crime Program, This MITRE C;orpo!^^on and the National Institute of 
Law Enforcement and Criminal Justice have* taken th? opportunity provid- 
ed by the large-scale implementation and evalifation of crime reduction 
projects in the eight Impact cities to examine the process ani^techniques of 
project-level evaluation. ^ ^ 

A major areYi of inquiry for the national level evaluation is the planning 
phase in the evaluative, process. Evaluation planning is therefore. being 
assessed in each of the Impa^ cities in terms of the organizational place- 
ment of evaluation responsibility, the completeness and adequacy of 
project-level evaluation plans (components), and the composition 6f staffs 
assembled lo implement these plans. The importance of the role played by 
Impact project evaluation components led to the development of a model 
and of revievy criteria for assessing them vVhich are presented here in the 
belief thai they can usefully serve practitioners and reviev/ers in the field. 

The model and criteria presented herein have evolved slowly over the 
course of the Impact Program. The insights gained from the review of the 
many evaluation plans developed by city and project evaluators have been 
inviOuable in this effo/l. 

The following^ paper is divided into foi^-sections. The firs^ (introductory) 
sectiorf describes current preoccupations with evaluation. The second sec- 
tion provides the reader with an understanding of the special conlejtt within . 
which the model and criteria were developed via a brief discussion of the 
Impact Program's evalu^^tiop effort. The third section presents the evalua- 
tion planning model along with a discussion of key steps in the evaluation 
planning process. The fourth section elaborates general guidelines regarding 
the use and applicability of the model and review criteria, and develops a set 
of questions which need to be addressed during the review of a project-level 
evaluation plan or component. - 
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1.0 INTRODUCTION 

The responsibility for providing certain social services has shifted over the 
last several decades from the domain of families, neighborhoods, and 
employers to the public sector. This shift has been accompanied by large 
outlays of federal monies to finance Jboth an increase in existing services and 
^ a wide range of nevy services in an effort to address perceived social 
problems. ^ 

The fact that these social problems persist despite these efforts and large 
expenditures is a continuing source of frustration for the policy-maker and 
citizen alike. Programs believed to be bold and innovative solutions to social 
jjroblejns have often failed to achieve vvha^^was expected of them. While ex- 
pectations v^ere somelirties unrealistic, ^^lous questions have nonetheless 
been raised about the process of program selection iind assessment and the 
adequacy of programmatic infjjj-mation available to guide this process. 

In response to these questions and to the information gaps which they 
reprejient, demands have increiisingly^been made ;ipon evaluation as a likely 
source for more data on the costs and benents of social programs. Evalua-, 
lion in this context i^ a process of accounting for-ihe expenditure of funds by 
examining wliat happens to a specific problem when money is.expended and 
services delivered to address that problem. Whereas previous accountability 
efforts focused upon how monies were spent or whether services were 
delivered, the question now being posed targets the effect such expenditures 
and services have on the problems they are deSigrred to address. 

Kxperience shows that the answers to these questions do not come easily. 
To date, there have been serious vyeaknesses in the range and quality of 
evaluative information generally produced. These weaknesses may be partly 
attributed to the newness of the effort and to the frequently post-hoc nature 
of many evaluations. To insure the collection of data needed to assess 
program activities and outcomes* adequately, evaluation plans must be 
developed prior to program implementation. Where these plans are either 
absent or unsatisfaj^tory, the chances for obtaining useful evaluative infor- 
mation appear to be greatly decreased. 

This paper is designed to help practitioners iJnd policy-makers increase 
ihcir chances for obtaining useful evaluative information by providing a 
model and a ^Gt of criteria for *Veviewing project-level evaluation plans. 
While the model and review criteria presented in this document were 
developed within the context of the LHAA*s High Impact Anti-Crime 
Program, they ^verc based upon an awareness of the difficulties involved in 
evaluating 'socilil programs generally. The model and criteria thus evolved, 
not from a special consideration of criminal juslig? programs, but rather 
from a broader perspective v/hich addresses measurement problems in a 
dynamic environment. It seems likely, therefore, that the model and review 
criteria may be applicable, as wall, to Evaluation efforts outside the criminal 
' Justice sector. 
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'2.0 EVAtUATION IN v:i>IE IMPACT PROGRAM 

The Impact Program, launched by the LaVv Enforcement Assistance Ad- 
ministration {LEAA> in 1972, was designed to address the t)roblem of 
street-crime and burglary i^ eight major U.S. cities; Impact, from the ou^ 
set, had a service, demonstration, and accountability orientation. It was 
.designed to reduce crime through the provision of services, demonstrate the 
utility of crime-oriented planning as a rational way to select these services, 
and implement program-wide evaluation as a means for assessing the extent 
to which tBese services actually improved targeted cri^ne problems- in the 
eight Impact cities. 

Evaluation has been incorporated into the Impact Program, at three 
different levels. The broadest leve!| addresses thci degree of Impact crime 
reduction. Data with \vhich to answer this questioi^are to be provided by a 
series of victimization surveys administered with thls/jupport of the Bureau 
of the Census. ' 

Evaluation will also take place at the national level and at the city level. 
At the national level, evaluation is designed- to assess various facets of the 
Impact t*rogram across the . eight cities. This effort includes an examination 
of the planning, implementation, and»evaluation activities of these cities as 
well as an overall assessment of program strengths and weaknesses, gjity- 
level evaluation v/ill include project-specific evaluations as well as a city- 
wide assessment of the effectiveness of broad strategies selected by each city 
to address their crime problems. Project evaluation efforts are designed to 
provide information about the activities and outcomes of specific anti-crime 
tactics. Here city evaluators are responsible for determining the extent to 
which crime problems targeted by a specific project improve in the manner 
originally anticipated. ' ^ 

"^e importance of project-level evaluations in the Impact Program can- 
not^be overemphasized. These evaluations provide information needed to 
assist decision-makers in allocating limited resources, to identify project 
operational areas in need of improvement, and to contribute to the body of 
knowledge essential for effective planning and proGlem-solving. The impor-^ 
^ tance of project-level evaluations is reflected in the LEAA requirement that 
each Impact-funded project be evaluated during the course of project 
operations. To insure the fulfillment of this requirement, the LEAA initially 
urged the development of project-specific evaluation plans (components) 
prior to project implementation. This latter requirer^nt provided the, im- 
petus for developing the evaluation planning process model and review 
criteria presented in the remaining sections of this document. 
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3.0 A PROJECT^LEVEL EVALUATION PLAN- 
NING MODEL FOR THE IMPACT PROGRAM 

Project-level evaluation components were intended to serve as 
"blueprints" for subsequent project evaluations. The LKAA expected these 
components to provide the foundation for evaluati«)n by furnishing: 

(a) a delineation of project objectives; 

^(b) evaluation measures; ^ 

(c) data re4tiirements; ^ 

(d) a data collection approach; 5 

(e) a data analysis approach; and 

(f) an evaluation reporting schedule. ^ 

While an assessment of project-level evaluation components in the Im- 
pact Program must therefore revolve around the six elements specined by 
LhA/^, these elements are nonetheless in^sufficient in themselves for an 
^ adequate pre-evaluation design.' Although they do define the skeleton of an 
evaluation component, they reveal little about the quality of that structure. 
* ' Moving hcyond these basic structural elements requires an understanding of 
the purpose of the structure and the role It plays in the evaluation process.^ 
A<; previously mentioned, an evaluation^omponent is needed to provide ' 
the basic blueprint for subsequent project evaluations. That is, it stould 
serve as a vehicle for defining, collecting and analyzing the data needed to 
assess the value of a.pai*licular anti-crime effort in terms of its stated aims. 
* Such value may be gauged by addressing three basic questions: 

{d\ Did the project actually implement the agtivities/deliver the services 
'which v/cre specified in the grant application? 

(b) Did the crime levels that the project was designed to reduce actually 
decline? \ 

(c) Is it reasc^able to attribute such 4mprovement lo fhe project's ac- 
tivities? 

If one accepts these three questions as legitimate foci for an evaluation ef* 
fort. It is then reasonable to assess evaluation plans in terms of their an- 
\ ticipated ability to insure the collection and analysis of the information 
] needed to answer these questions. It is in this ci)nte?;t that the follov/ing 



model and review criteria have been developed. 



3.1 Tho fivalyatldii Piannlfig Proeosa 

The real starting point in ^he evaluation planning process (depicted in 
f igure I, see Page 4) is the identification of a specific crime problem. The 
nature and extent of this problem drive the remaining steps in the process. 
Project activities develop from the need to implement a particular anti- 
crime strategy believed to combat the pre-identified crime problem. These 
activi^es must therefore be logically linked to project outcome goals and , 
objectives which, in turn, refiect the desired changes in the identified crime 
problem. The remaining interdependent steps in the evaluation planning 
process, from the delinea^n of activity, intermediate, and outcome objeg- 

NalimKifTnoTitFtc Memorandum, (iuidelincT for Kccional Office Kcvievv of livaluation 
Q ( timponenti of Impatt ( ity 'Project Propo^ah. 2^ February 197 V 
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lives through the specification of measures, data collection and analysis 
procedures, constitute the basic foundation for assembling evidence to sup- 
port subsequent inferences about linkages among project activities and out- 
comes. ^ * 
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3.2 Projeot Objegti vos 

An important step in the project-level evaluation planning process in- 
volves determining what the project expects to accomplish not only in terms 
^•of its effectiveness, or outcome^, but also in terms of its acti^^ities. Activity 
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objectives specify the type, range, and amount of ^services to be delivered, * 
the target area/target population which v/ill receive these services, and the 
manner in which these services are to be delivered. Outcome objectives in- 
dicate the kind and extent of improvement anticipated vig-a-vis the iden- 
tified crime problem. Additionally, these objectives need to specify in quan-/ 
titative terms the precise level of improvement expected, as well as ihp 
amount o£ time deemed necessary to achieve the outcome objectives. / 
In some instances, however, the re^ll improvements or ultimate outconles 
the project is designed to produce may not be measurable on a short-term 
basis. For example, a project targeting recidivism"}nay seek to reduce the^ 
recidivism rates of serious, adult offenders by providing intensive counseling 
apd educational services in a community-based treatment facility. Since the • 
target population is physically confined during the period of project treat- 
ment, and hardly in a position to recidivate, it may take several years to 
determine the extent to which the project has met its primary outcome ob- 
jective — recidifjttn reduction among serious, adult offenders,- In the in- 
terim, inform^Kn will be needed which allows evaluators and decision^ 
makers to gauP how well the project is progressing in terms of its stated 
aims. To provide this information, intermediate objectives which are 
presumably linked to the ultimate desired outcomes need to Be formulated. 
These objectives specify a set of outcomes wlych are assumed to facilitate or 
^ reflect the achievement of the desired long-term improvements in the 
targeted probkm. 

In the earlier example of the recidivism reduction project,tmprovements 
in client educational achievement levels or client feelings of self esjeem . 
might be used as intermediate project objectiv&. The expectation or 
assumption here is that a client^s level of educational achievement and/or 
feelings of self esteem will be important determinants of future involvement 
in crimrnal behavior. Assuming this.to be a reasonably; expectation, attain- 
ment of these intermediate objectives provides a basis for determining how 
well the project is progressing towards its- ultimate outcome objective. 
4 Unfortunately, there is often little evidence about presumed linkages 
among activity, intermediate, and outcome objectives. The mOvSt reasonable 
approach, given the need for timely evaluative information and existing 
kno.wledge gaps, is therefore to delineate the most logical set of activity, in- 
termediate, and outcome objectives, keeping in mlnd'the tentative nature of 
the linkages among them. When these objectives are in fact logically linked 
together they provide a coherent conceptual framework foq the development 
of internally consistent evaluation methods, instruments, and tools. This in- 
ternal consistency and tlie confidence it generates in the method of evalua- 
tion helps the cvaluator io better assess the soundness of the assumptions 
underlying the project\ objectives as well as the<txtent to^ which these objec- 
tives are being met. 

3.3 Moasyres 

After delineating activi^fTrntermediate and outcome objectives, valid 
measures are developed mv use in the project evaluation. Measures bridge 
the gap between an objective and the data required to assess whether or not 
y . specific objective has met. That is, they define the observable 
§^ . .5 .11 



.behaviors or criteria which ultimately ^erve as the basic body of evidence 
underlying conclusions or inferences about project/objective attainment. 

Bridging such a gap requires translating key aspects, and dirnensians of 
tbe project into criteria which aPe noi only measurable, but demonstrably 
vaW in that they effectively measure achievement of project objectives. The 
ba-j^ ideas tte^r()ject, its aims, and important side-effects (such as crime 
displacenient)^eed to be captured and accounted for in the proposed 
measures in ilpisr that a comprehensive assessment of projecj^cttievemenls 
can take place. These measures must thos be validMndicafors of the con- 
cepts, aims, and side-effects they are designed to reflect, Drnd the key ques- 
tion here is whether the proposed measures ^teally measure ^lat they are in- 
^ ' tended \o m'easu re. ^ 

Measures must alsy] be operatn)j^lly defined in thcf evaluiition plan. 
These opei-ational definitions specify thcseti)f ccmditimis.or eyents which 
signal the presence ()r absence of the activity or out<;mne being measured. 
\ ox example, educational achievement is frequeruh^ used as a-measure or in- 
' dicant "of rsociyl adjustment in 'rehabilitation^oRcts ' targeting juvenile 
offenders, Assuming thiJ to be a valid jmeas^pc^^ does" the evaluator 
know vvhich juvenilch are in fact incrtwt?!^ tli^i*^ level of e9uca{,ionaI 
achievement? What is needed is /an operational definition of educational 
achievement >^vhlch specifies thi)se behaviors, activities, or events which 
allow liie cVlftuafc)r to' clearly discriminate achievemeat levels among " 
' juveniles. Iti this case, the successful completion of cQurse v/ork, passing 
' grades, or grade-level prom()tions mjght he among the bdiaviors or events 
used to operatu)nally define ediicational achievement, thus providing the 
evaluator with a more pfccise basis for measuring one type of improvement 
in the level of s()cial adjustment am()ng juvenile <)ffenderSf 

Also of importance is the sensitivity ()f the evaluation measures and their 
corresponding operational definitiims. Pr()posed measures may b^ loo crude 
^ t() reveal the nature and extent of changes wfiich the project may create both 
in terms ()f its activities and outcomes. That is, the specified unit of measure, 
must be ^^ble to refiect changes which may be ()ccurring relativt?, to tlii 
targeted problem. In the earlier example of educational achievement among 
luvenile offenders, th^ use ()f grade-level proirn)tu)ns or graduations tfrom 
high school t() differentiate achievers frdm non-ach4evGrs may result m mis- 
leading conclusions about project outcomes. These tv;o measures are, in a , 
sense/to,o gross tw revQj^l in^portajit changes whidi may*be occurring ^imong 
project clients those juveniles vvho had rarely 'completed or passed a'^ 
course prior to pr<iject participation, the successful completion of several 
courses w()uld certainly indicate an increased level 0f educational achieve- 
ment If, however, 'the evaluat(>r relies striclly ()n grade prom()tions or high 
V*h<H)l grad\jations\n) unique indicants of educational achievement, these 
improvements might easily gir unnoticed, * 
• . I bus, 'the viilidity of the proposed measures and the sensitivity ()f their 
corresponding operational definitums are critical to the evaluation effort. il^) 
C()nceft, they allow the evaluator to'assemble evidence to support con- 
. . elusions about the extent U) which project objectives have been met. 
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3.4 Evaluation Research Desfgn/Methodology 

Ofjce measures have beeij>defincd, an evaluation^ research di^sign needs to 
be developed tp provide a method for identifying changes in the targeted 
problem and, at the s^inrie time, allow the evaluator to determine whether 
'these gjbserved changes in outcoune* measures ©an reasonably be attributed 
10 ibeprgjea's activities rather than to other external factors or to chance. 

ln,t)rder To .identify changes or differences^in the targeted problem, §ome 
basis for-compariso'h is essential Ideally; the evaluator would like to Hise 
outcome measures tSken from a randomly selected control area/gri)up dur- 
ing the perioH ofproje^ qperations^as tlje basis for comparison. This type of 
comparison' guar^ritees that the effects of outside influences wil^npt? 
systematically bias observed changes in outcome measures. In the case oT a 
project designed to redupe recidivism among juvenile offenders, for exam- 
{jle, the random assignment of offenders ,]to the project treatment group and 
to a non-treitment (control) group allows the evaluator to assume that fac- 
tors which may affect recidivi»m ;rates, 'such. as ctit^t criniinal^history or 
age, will not systematically bias tjie recidivism rates observed in either the 
treatment or control groups. I^^i the absence^of systematic biases in observed 
outcomes, the ej-^aluator i^ in a better position to say that ob3erved 
differences in recidivism rates between treatment" and non-treatment groups 
are attributable to the project's activities. 

When control through randomization is not feasible, other approaches 
must be usedao examine the relative impact of the project and of other in- 
fluences upon The observed changes in the measures. Control through the 
use of conrparison areas/groups matched to the targeted area/group on the 
basis of selected characteristics is one alternative, as is the use of statistical 
techni(^es which may factor out estimated influences which are expected to 
affect outcome measupes during the project period. When these alternatives 
are used, the validity of the findings obtained will be directly rdated to the 
evaluator's ability to identify and discriminate among those characteristics 
or factors unrelated to projectilactivities which may infhiente the buto>me 
measures being examined. For example, when juvenile offenders a%nbt 
randomly assigned to project treatment and non-treatment groups, the 
evaluator may .attempt to identify a set of characteristics, such as age, 
criminal history, educational level, which are assumed to affect recidivism 
levels in the treatment group. These characteristics would then guide the 
selection or identification of another group of juvenile, offenders whose 
recidivism levels during the period of project operations would be compa^^pd 
to those observed in the project treatment group. Differences in refeidivism 
levels o^^erved among these two groups cannot be blindly accepted, 
however, as estimates of project effects. Rather, it must be recognized that 
the degree of correspondence between observed differences and project im- 
pact depends uipon the validity of the assumptions made in selecting the set 
of characteristics used to develop the comparison group. Thus, the extent to 
which the evaluator can identify significant characteristics or factors greatly 
affects the degree to which observed changes are indeed attributable to pro- 
ject activities. 

The evaluator's ability to do so is likely to be rather modest, however, 
^9^.en tlie limited state of knowledge about the dynamics of complex social 
lie 7 1 Q 



problems such as crime. This* knowledge, nonetheless, provides a basis for ,^ 
examining, the validity of assumptions underlying the selection and use of a * 
particular basis of comparison in the evaluation effort.* 

3«5 Data Collection Plan 

Project objectives, measures, and the research design together make data 
coflection a meaningful operation: they^ define the kinds of data which are 
needed and the mariner in which they witj subsequently be aggregated and* 
analyzed to provide information, ^bout project activities 'and outcomes. 
; Without reliable data, the evaluation plan is like a recipe v^hich Has either 
not been tried because the ingredients ai^e unavailable or has proved un- 
successful because the ingredients used were ofjow quality or were 
questionabte substitution^. Developing a mechanism for obtaining reliable' 
data is therefore a vital step in the evaluation planning pflrocess; > 

BasicaHy two types oCdata are needed for the evaluation effort. The first -"^^^ 
includes those data elements needed to construct ^jroject activity and out- 
'Come measures. These data elements, previously identified in the process of 
specifying evaluation measures,^|p.hfTi the basis for making conclusions 
_^ aiiQut-tl^exxtent to which project ofe^i^^i 

consists of those data elements need||to implement the control feature of 
the research design (that is, data on s|j|cted characteristics or factors which 
^. will be controlled for through either ajf|atching process or somfe method of 
analysis). ThesCdata elements, identifilsi^ in the process of selecting a h^^sis 
for comparisqn, are crucial to the evaluator's efforts to determine whether 
observed changes in outcome measures|pn reasonably be attributed to the 
project's activities. In conjunction wi^^jie another, these two types of data 
provide the raw ingredients needed to as^|ss project impact on the targeted 
problem.) 

3.5.1 Data sources. Developing £f dita collection approach involves^ 
identifying potential data sources, constructing data collection instruments, 
and in some case§, specifying the sampliiV apjjroach and the population 
from which data will be collected. The car^ identification of data sources 
provides the opportunity to gauge whetherlr not the data elements needed 
to develop the measures and implement thdresearch design will in fact be^ 
available. When data gaps are identified at^an early stage in the process, 
necessary modifications in the evaluation plald^can be made prior to its full 
implementation. This helps to insure that the^ubsequent collection of data 
wtn be useful and will result in a proper execu^^on of the,evaluation design. 

i 3.5.2 Data forms. Data collection, instr^^ients arc constructed to 
provide a method for recording and categorizihfe/needei^ data, Ultimately, 
the data collected arc only ^s good as the manni^ in which they are record- 
ed. Where data are categorized in a fashion which makes it impossible to 
differentiate client sub-group populations or different types of project ac- 
tivities, useful information may be hopelessly lost. It is thus important to * 
develop data collection procedures and forms which specify categories that 
are mutually exclusive. Additionally, data collection procedures and forms 
O S|hould clearly correspond to the range and level' of data required for the 
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evaluation effort. If information is needed at a client-specific level, data 
forms whi:ch encourage the recording of strictly aggregate, grmxp data are 
clearly inadequate. Similarly, if information on client socio-economic 
^ background is needed, provisions sbpuld be made so that this information is . 
recorded on "tjie data collection forms. 

When it is infeasible t6 collect data from the entire population of interest,, 
plans for evaluation may include the collection of data from a sample or 
sub-group of the population^. Here, the criteria guiding the selection of the 
sample and the size- of the sample which is to be used must be carefully con- 
sidered in terms of their ability to generate an unbiased, representative sam^ 

, pie. For example, a project targeting bjirglary problems in'a high crim^e area 
may seek-lo increase community awareness of the importance of preventive 
devices such as locks and-t)urglary alarms< In this cafse, the evaluation plan - 
(nay include the collection of attitude da^a from a sample of high crime area ; 

J residents. In order to get a ffiir reading of citizen attitudes towards preven- 
tive devices the evaluator must select the sample in such a way that the in- 
formation collected is representative of the population of interest; in this 
case, higK crime area residents. Additionally^ the sample must be large 
' enough to justify making conclusions about the population as a v/hole. 
Biases or lack of representation can most easily be avoided by randomly ' { 

selecting the^ sample. Other appi*oachcs, such as a stratilted sampling ap- 
proach, are* acceptable when the criteria or characteristics used to stratify 
the sample appear to be reasonable. 

To further insure 4he collection of needed data, responsibilities far data f 
collection and validation must be clearly specified prior to the implementa- / 
Mion of the evaluation plan. Too often, confusion oveP data collection 
responsibilities has resulted jn a failure to collect dat.a essential for the 
' evaluation effort. Similarly, failure to check data for inconsistencies in the 
recording of information have thwarted an otherwise well-designed evalua-' 
tion effort. Thus, the data collection approach developed in the ©Valuation 
plan must include the specification of the data collection responsibflities and 
validation procedure^, as well as the identification of the sources, in- 
struments, and sample approach which will be used to collect needed data. 

3.6 Project Monitoring and Evaluation Reporting 

An evalu^ition plan must also specify a system for ifionitoring projegt ac- 
tivities and reporting project outcomes. Project monitoring during the Ijfe of 
the project provides a mechanism for identifying operational weaknesses 
which may ultimately affect project outcomes and/or preclude the collec- 
tion of information needed for interim evaluation reports. These reports 
provide an important feedback mechanism which affords evaluators the op- 
portunity to test their original evaluation plan and make modifications 
which v/ill facilitate the subsequent production of information useful for 
decision-making purposes. To insure the existence of this self-correcting 
process, each evaluation plan should include a discussion of the monitoring 
system, and of the frequency with which evaluation reports will be written 
and disseminated. 

lb 
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4.0 REVIEW OF EVALUATION PLANS 
^ 4.1 A Note on Assessing Evaluation Plans ^ 

While the evaluation planning pr4)cess has been discussetfas a series of 
sequential activities, it should be kept in mind that these activities are really 
_ ^ part of a complex, iterative process. Changes or modificatmns in any step in 
* . the process usually have an impact on the other steps. For example, a 
resGoping of project outcome objectives necessarily affects the applicability 
\ of previously defined measures and data collection sti'ategies. Similarly, 
data constraints (encountered during or after the development of the evalua- 
fy< i\on plan) limit not only the type and range of measures which may be used, 
but also the type of evaluation design/methodology v/hiqh is appropriate for 
' linking project activities and outcomes.Thus, data considerations — like all 
^ of the activities in Figure 1 (Page 7^ feed into the evaluptiv'^ process in af 
cyclical vyay. ( 

This interdependcncy means that the initial evalyatior^lan v/hich is ^ 
prepared and reviewed is rarely executed in its original form. For this 
• reason, the use of the review questions, presented below should^likewise be 
j viev/ed ^s an it-erative process, to.be repeated as modifications are required 

in the original evaluation plan. Further, while the model and review 
questions provide a viable'method for assessiYig the adequacjp" of specific 
aspects of the evaluation plan, the overall logicjof the plan and the extent to 
which elcmeiTts are logically linked together are not specifically addressed. 
The logical consistency of the overall evaluation plan is therefore an ad- 
ditional and overriding issue which must be raised and addressed in light of 
the nature of the project, the limitations of*the research context, and pur- 
pose of the eviiluative effort. 

4.2 Evaluation Review Questions 

As indicated earlier, an. evaluation plan provides the foundation for 
assembling information needed to assess linlcagcs among project activities 
and outcomes. The soundness of this foundation ca^i be assessed by review- 
ing each of its elements in terms of several basic questions. These questions, 
listed betow, constitute the criteria developed to review project-level evalua- 
tion plans or components. <^ 

I. Project Objectives 

Questions to ask about project objectives when reviewing an evalua- 
. tion plan include: ' & * 

(a) Are the basic idetis of the project adequately translated into 
measurable goals and objectives? 

(b) Are activity objectives delineated which specify: 

• type of services t6^e provided; 

• range or scope of services to be provided; 

• quantity ofservices to be provided; and 

• service recipients (e.g., target population, target area)? 

y (c) Do the intermediate objectives which have been delineated 

specify: 

- ^ • kind and extent of improvement anticipated ^ 
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• aquanliTied level of expected.achievement 

• the period of time deer^d necessary to achieve objectiv^es? 

(d) Axe outcome goals/objectives delineated which specify: 

• the kind and extent of improvement anticipated vis-a-vis the*^ 
identified crime problem; 

• a quantified level of expected achievement, and 

0 the period of timl^ deemed necessary to achieve goals/ 
objectives? 

(e) Are activity objectives, imerm^diate objectives, and outcome 
goals/objectives logically linked together? 

(0 Are the activity,, intermediate, and outcome objectives realistic 
in terms of expected levels of achievement? 

2. Evaluation Measures " ^ 

Questions to ask about evaluation measures v;hen reviev;ing an eval- 
uation plan inchide: . . 

(a) Are the basic ideas of th$ program adequately tramslated into 
the proposed .measures? In other- v;ords, are key aspects/ 
dimensions of project goals/objectives tapped by the proposed 
measures? Are important side-effects (such as crime displace- 
ment or system changes) captured and acc6unted for? 

(b) Do the proposed measures appear to be valid indicators of key 
project concepts and objectives? In other v/ords/do the measures 
really measure v;hat th^y are intended to measure? 

(c) Are the measures adequately operationally defined? 

(d) Are the propo^d rtieasures sensitive enough to show the nature 
and extent of charfgl^ which the project i^^ expected to create 
both in terms of activities and outcomes? That is, can the specified 
uhit of measure reveal changes v;hicH may be occurring in the 
targeted problep? 

S. '^Evaluation Research Design/ Methodology 

Questions to ask about the evaluation research design/methodology 
when reviev;ing an evaluatidn plan include: 

(a) Is some basis for comparison specified in the evaluation com- 
ponent? 

(b) Is the basis for comparison sufficiently described to permit a 
critical assessment of its adequacy? 

(c) Does the evaluation research design/methodology provide 
controls '(either through the treatment ass^gmment process or 
collection and analysis of data) for: 

a selection biases; 

e inappropriate treatment selection criteria; 

• impact of natural phenomena (seasonal variation, maturation); 

• Impact of events outside the project v;hich could blunt or 
exaggerate measures of project outcomes?. 
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4. Data Collection Plan 

Questions to ask aboutjlhe'data collection plan when reviewing an 
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evaluation plan include: 

(a) Are mechanisms for "Collecting required data clearly specined 
in terms of: ' . 

o sampling approach; . * * 

^ sample si/e; 

o data collection forms; * 
o data sources; , ^ 

0 responsibility for data collejction*^ 
'^ 0 procedures for data validation?! - 

(b) Are the data collection forms adequate mechanisms for col- 
. lealing the range and level of data required to implement the 

research/methodology? ^ 



Evaiuation Reporiing Schedules 
Questions to ask about evaluation, r6p6rting schedules^ when review- 
fng an evaluifl'kxn pt^an include: 

(a) Is^an evaluation reporting^schedule included in the plan? 

(b) ^ Is the schedule reasonable in light of: 
^ ♦ project duration, and 

• nature of project? 

4.3 ^Use of Review Questions , 

The review questions presented above provide a method for systematical- 
ly assessing project-level evaluation plans. By using these questions, missing 
elements in these plans can be quickly identified. Similarly, inadequacies in 
ihe substance of those elements addressed in the evaluation plan can he pin- 
pointed vis a vis the revgew question procedure. Early identification of gaps 
and inadequacies allows the evaluator to make modifications- which will 
facilitate the subsequent^producticfn of useful ev^iluative information. 
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